PCT 



WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




(51) Internationa] Patent Clas 


siflcation 6 : 




(11) International Publication Number: 


WO 96/10888 


H04N 5/50 




Al 












(43) International Publication Date: 


11 April 1996 (11.04.96) 



(21) International Application Number: PCT/US95/ 10662 

(22) International Piling Date: 22 August 1995 (22.08.95) 



(30) Priority Date: 

08/316,503 



30 September 1994 (30.09.94) US 



(71) Applicant: INTEL CORPORATION [US/US]; 2200 Mission 

College Boulevard. Santa Clara, CA 95052 (US). 

(72) Inventors: ADAMS, Robert; 17606 Cardinal Road, Lake Os- 

wego. OR 97034 (US). WILLIAMS, David, M.; 2731 S.W. 
Bucharest Court, Portland, OR 97225 (US). RICHARD- 
SON, John; 2748 N.E. 19th Avenue, Portland, OR 97212 
(US). PERRY, Burt; 8145 S.W. Barnard Drive, Beaverton, 
OR 97007 (US). 

(74) Agents: SALTER, James, H. et al.; Blakely, Sokoloff, Taylor & 
Zafman, 7th floor, 12400 Wilshire Boulevard, Los Angeles, 
CA 90024-1026 (US). 



(81) Designated States: AM, AT, AT (Utility model), AU, BB, BG, 
BR, BY, CA, CH, CN, CZ. CZ (Utility model), DE, DE 
(Utility model). DK, DK (Utility model), EE, EE (Utility 
model), ES, FI, FI (Utility model), GB t GE, HU, IS, JP, 
KE, KG, KP, KR, KZ, LK, LR, LT. LU, LV, MD, MG, 
MK, MN t MW, MX, NO, NZ, PL, PT, RO, RU. SD, SE, 
SG, SI, SK, SK (Utility model), TJ, TM, TT, UA, UG, UZ, 
VN. European patent (AT, BE, CH, DE. DK, ES, FR, GB 
GR, IE, IT, LU, MC. NL, PT, SE), OAPI patent (BF, BJ, 
CF, CG, CI, CM, GA, GN, ML, MR, NE, SN, TD, TG), 
ARIPO patent (KE, MW, SD, SZ, UG). 



Published 

With international search report. 



(54) Title: CONTENT PROGRAMMER CONTROL OF VIDEO AND DATA DISPLAY USING ASSOCIATED DATA 




(57) Abstract 

An interactive video system is disclosed that processes a video data stream and an associated data stream corresponding to the video 
data stream. The interactive video system displays a video image defined by the video data stream on a display device (50) and performs 
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CONTENT PROGRAMMER CONTROL OF VIDEO AND DATA 
DISPLAY USING ASSOCIATED DATA 

Background Of The invention 
1. Field Of Th e Invention: 



The present invention pertains to the field of computer 
systems. More particularly, this invention relates to the coordination 
10 of video and audio data streams using associated data streams to 

enable content programmer control of display and selection functions 
for a video system. 

2. BACKGROUND : 

15 

In prior television systems, a video content programmer 
typically controls the entire video display screen by selecting the video 
programming, the presentation of the video programming, and the 
programming flow. Typically, such a content programmer maintains 
20 total control over the coloring, the style, and the presentation of 
images on the video display screen. 

In contrast, the presentation of video on a computer system is 
typically limited by an operating system's styles and conventions. A 
25 typical operating system for a computer system imposes a particular 
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display style that confine programs executed on the computer system. 
In the case of video windows on such a computer system, a content 
programmer is usually limited to the standard look of the particular 
operating system and may not vary too far from the standard look. 

5 

However, as such computer systems become more commonly 
used for the presentation of consumer items, content programmers 
may seek more control over the look and feel of computer system 
video. Such control would enable content programmers to select a 

10 look and feel closely identified with the products being displayed. 

Such control would require that the content programmer control the 
video information as well as other elements on the video display 
screen including colors, logos and selection button placement. 
Unfortunately, such control over the look and feel of video displays 

15 on a computer system is generally unavailable to content 
programmers on prior television receiving systems. 

Prior television receiving systems merely provide mechanisms 
for including data associated with an incoming video signal. Such 
20 associated data is typically received during vertical blanking intervals 
of the incoming video signal. In such prior systems, the functions of 
the associated data in relation to the incoming video signal is usually 
fixed according to the design of the television receiving system. 
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For example, some prior television receivers include a 
mechanism for decoding and displaying closed caption information 
that corresponds to the incoming video signal. In such a system, the 
closed caption associated data of the incoming video signal is usually 
5 decoded and displayed with video characters superimposed over the 
normal incoming video scene. 

Other prior video receivers decode the associated data of the 
incoming video stream and transfer the decoded data through an 
10 input/output port of the receiver. Typically in such systems, the 
function of the input/output port is fixed according to a 
predetermined design of the receiver. Unfortunately, such systems 
impose fixed and inflexible functions to video content programmers 
where the receiving device is a computer. 

15 

In addition, some prior video receiving systems provide 
rudimentary interactive video functions. In such systems, selection 
regions of the video display are typically encoded into the incoming 
video signal. Such selection region information usually specifies 
20 areas of the video display that correspond to a predetermined 

selection function according to the content of the incoming video 
signal. The video receiver in such prior systems typically includes a 
pointing device that enables a user to position a pointer within a 
selection region and select the corresponding function. 

25 
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Unfortunately, such prior interactive video systems provide 
only limited selection and control functions for interactive video such 
as specifying regions for picking underlying video objects. Typically, 
such prior systems do not enable video content programmers to 
exercise flexible control over an entire range of interactive video 
functions including the sizing and placement of video objects over 
the entire display. 
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SUMMARY AND OBTECTS Of THE INVENTION 

One object of the present invention is to provide flexible 
content programming control in an interactive video system. 

5 

Another object of the present invention is to enable a content 
programmer to create a video display screen from a programming 
studio and to flexibly, control the area around the video display 
including the placement of text, the definition and placement of 
10 graphical objects and to provide download information to the 

computer system and place buttons on the video display to select the 
downloaded information wherein the additional display information 
is in sync with the video and audio information. 

15 Another object of the present invention is to provide an 

interactive video system controlled by an associated data stream 
corresponding to a video data stream. 

Another object of the present invention is to provide an 
20 interactive video system wherein the associated data stream 

determines placement of a video display window on a display device. 

Another object of the present invention is to provide an 
interactive video system wherein the associated data stream 
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determines placement and functionality of selection regions that 
correspond to a video image in the video display window. 

A further object of the present invention is to provide an 
5 interactive video system wherein the associated data stream 

determines the placement and pixel content of graphical objects on 
the display device that correspond to the video image. 

Another object of the present invention is to provide an 
10 interactive video system that employs the vertical blanking intervals 
of the video frames of a video signal or chrominance, sideband, or 
audio sub carrier signals to transfer the associated data stream. 

Another object of the present invention is to provide a video 
15 system that employs packetized digital data streams to provide a video 
stream, an audio stream and a command and control associated data 
stream. 

These and other objects of the invention are provided by a 
20 video system that receives a video data stream and an associated data 
stream corresponding to the video data stream via a broadcast 
receiver, a CATV link, or a packetized digital communication link. 
The video system displays a video image defined by the video data 
stream on a display device and performs command functions 
25 specified by the associated data stream. 
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The command functions include commands that specify 
parameters that determine an area on a display surface of the display 
device for placement of a video display window that contains the 
5 video image. The command functions include commands that 
specify parameters of graphical objects that corresponds to the video 
image and commands that specify pixel data for the graphical objects. 
The command functions include commands that specify parameters 
for placement of selection windows that correspond to the video 
10 image and commands that specify functions performed if a user 
selects the selection windows. 

One embodiment of the present invention is employed in a 
computer enhanced video shopping system. For this embodiment, a 

15 portion of a video display screen is allocated to the display of an 

incoming video program. The remainder of the video display screen 
contains information that describes items being displayed along with 
manufacturing logos for the items being displayed on the video 
display. The remainder of the video display screen includes selection 

20 regions or buttons to enable a viewer to select more detailed 

information regarding the items being displayed and related items for 
sale. In addition, the remainder of the video display contains 
additional text and graphics that provides information on future 
items for display. 

25 
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Other objects, features and advantages of the present invention 
will be apparent from the accompanying drawings, and from the 
detailed description that follows below. 
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Brief Description of the Drawings 

The present invention is illustrated by way of example and not 
limitation in the figures of the accompanying drawings in which like 
5 references indicate similar elements, and in which: 

Figure 1 illustrates an interactive video system for one 
embodiment that receives data streams through a satellite receiver, a 
cable television (CATV) receiver, or a television broadcast receiver; 

10 

Figure 2 illustrates a computer system for one embodiment 
including a graphics display subsystem that drives a display device, a 
data modem that receives data streams, and an audio subsystem that 
drives a speaker; 

15 

Figure 3 illustrates the data modem for one embodiment which 
comprises a data selector, a video queue, an audio queue, and an 
associated data queue; 

20 Figure 4 illustrates a digitized representation of an analog video 

signal received by the data selector from either the CATV receiver or 
the television broadcast receiver for one embodiment; 
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Figure 5 illustrates a packetized digital data stream received by 
the data selector for one embodiment which illustrates the format of a 
video packet, an audio packet, and an associated data packet; 

5 Figure 6 illustrates the software architecture for the computer 

system for one embodiment which includes a data stream manager 
and an operating system; 

Figure 7 is a flow diagram illustrating the functions of the data 
10 stream manager for one embodiment, wherein the client runtime 
manager processes the video, audio, and associated data streams to 
perform interactive video functions. 

Figure 8 shows an example video screen that can be created 
15 with the technology described herein. 
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DETAILEP DESCRIPTION 

Figure 1 illustrates an interactive video system for one 
embodiment* The interactive video system comprises a computer 
5 system 10, along with a display device 12, a keyboard 20, a mouse 
pointing device 22, and a speaker 24. The interactive video system 
further comprises a satellite receiver 14, a cable television (CATV) 
receiver 16, or a television broadcast receiver 18. 

10 The satellite receiver 14 enables reception of packetized digital 

data streams over a satellite link. For one embodiment, the incoming 
packetized digital data streams received by the satellite receiver 14 
conform to the motion picture engineering group (MPEG) video 
transport standard. The packetized digital data streams received by the 

15 satellite receiver 14 include video data packets, audio data packets, and 
associated data packets. The satellite receiver 14 transfers the received 
digital data stream packets to the computer system 10 over a 
communication line 30. 

20 The CATV receiver 16 enables reception of an analog or digital 

video signal over a coaxial transmission line 28. The analog video 
signal over the coaxial transmission line 28 comprises a CATV cable 
television link. The CATV receiver 16 digitizes the incoming analog 
video signal and transfers the corresponding digitized video signal to 

25 the computer system 10 over the communication line 30. 
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Alternatively, the CATV receiver 16 transfers the incoming analog 
video signal to the computer system 10 over the communication line 
30. 

5 The television broadcast receiver 18 enables reception of an 

analog video signal transmitted over the air through an antenna 26. 
The television broadcast receiver 18 receives and demodulates the 
incoming broadcast analog video signal, performs analog to digital 
conversion on the analog video signal to generate a corresponding 

10 digital video signal. The television broadcast receiver 18 transfers the 
incoming digital video signal to the computer system 10 over the 
communication line 30. 

For one embodiment, the computer system 10 receives the 
15 packetized digital data stream from the satellite receiver 14 over the 
communication line 30. The computer system 10 extracts the video 
data packets from the incoming packetized digital data stream and 
generates a corresponding video image in a video window 40 on a 
display surface 50 of the display device 12. The computer system 10 
20 also extracts audio data packets of the incoming packetized digital data 
stream on the communication line 30 and generates a corresponding 
analog audio signal to drive the speaker 24. 

The computer system 10 extracts associated data packets of the 
25 incoming packetized digital data stream on the communication line 
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30 and decodes the associated data packets according to a predefined 
video command and control protocol. The associated data packets 
received over the communication line 30 specify a selection region 42 
on the display surface 50. The incoming associated data packets specify 
the position of the selection region 42 on the display surface 50. The 
associated data packets on the communication line 30 also specify a 
graphical display icon 44 for the display surface 50. In addition, the 
associated data packets specify the graphical content of the icon 44 as 
well as the positioning of the icon 44 on the display surface 50. 

The incoming associated data packets received over the 
communication line 30 specify an information display window 46 on 
the display surface 50. The associated data packets specify a position 
for the information display window 46 and the video window 40 on 
the display surface 50 as well as the display content of the information 
display window 46. 

For another embodiment, the computer system 10 receives a 
digitized representation of the incoming analog video signal from 
either the CATV receiver 16 or the television broadcast receiver 18 
over the communication line 30. The incoming digitized video signal 
comprises a series of video frames each comprising a series of video 
scan intervals and a vertical blanking interval. The computer system 
10 extracts the video information for the video window 40 during the 
video scan intervals of each received video frame over the 
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communication line 30. The computer system 10 extracts the 
corresponding audio information for the speaker 24 during the video 
scan interval of the incoming video data frames. 

5 In addition, the computer system 10 extracts associated data for 

the incoming video signal during vertical blanking intervals of the 
incoming video data frames. The computer system 10 decodes the 
extracted associated data according to the predefined video command 
and control protocol. The associated data specifies the parameters for 

10 the selection region 42. The associated data also specifies the position 
and content of the graphical display icon 44. The associated data 
specifies the parameters for the information display window 46 and 
the video window 40 as well as the display content of the information 
display window 46. 

15 

Figure 2 iUustrates the computer system 10 for one 
embodiment. The computer system 10 comprises a processor 52, a 
memory subsystem 54, a graphics display subsystem 56. The computer 
system 10 further comprises a data modem 58, a disk drive 60, an 
20 audio subsystem 62. The processor 52 communicates with the 
memory subsystem 54, the graphics display subsystem 56, the data 
modem 58, the disk drive 60, and the audio subsystem 62 via a system 
bus 51. 
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The memory subsystem 54 provides memory storage areas for 
an operating system, application programs, device driver programs, 
and associated data structures for the computer system 10. For one 
embodiment, the memory subsystem 54 comprises a dynamic random 
5 access memory (DRAM) subsystem. The disk drive 60 provides large 
scale mass storage to the computer system for programs and associated 
data structures. 

The graphics display subsystem 56 performs graphics rendering 
10 functions and includes a frame buffer and associated circuitry to drive 
the display device 12. The graphics display subsystem 56 enables 
software executing on the processor 52 to generate display windows on 
the display device 12 including video display windows. The graphics 
display subsystem 56 also enables user selection functions from the 
15 display surface 50 the display device 12 via the mouse pointing device 
22. 

The audio subsystem 62 enables software executing on the 
processor 52 to render audio data and includes circuitry that drives the 
20 speaker 24. The audio subsystem 62 receives digitized audio data over 
the system bus 51 and generates a corresponding analog audio signal 
to drive the speaker 24. 

The data modem 58 receives incoming digital data streams over 
25 the communication line 30. For one embodiment, the digital data 
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streams received over the communication line 30 comprise 
packetized digital data streams from the satelUte receiver 14. For 
another embodiment, the digital data stream received over the 
communication line 30 comprises a digitized representation of the 
analog video signal received either by the CATV receiver 16 or the 
television broadcast receiver 18. 

The data modem 58 enables software executing on the processor 
52 to extract the video data, the audio data, and the associated data 
from the incoming digital data on the communication line 30. The 
software executing on the processor 52 thereafter distributes the video 
data for display on the display device 12 through the graphics display 
subsystem 56. The processor 52 distributes the incoming audio data to 
the audio subsystem 62 to drive the speaker 24. The processor 52 
extracts the associated data through the data modem 58 and performs 
the interactive video functions specified by the associated data. 



Figure 3 illustrates the data modem 58 for one embodiment. 
The data modem 58 comprises, a data selector 76, a video queue 70, an 

20 audio queue 72, an associated data queue 74 and an address filter 75. 
The data selector 76 receives incoming digital data streams over the 
communication line 30. The data selector 76 extracts the incoming 
digital video data received over the communication line 30 and 
transfers the incoming digital video data into the video queue 70. The 

25 data selector 76 extracts the incoming digital audio data over the 
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communication line 30 and transfers the incoming digital audio data 
into the audio queue 72. The data selector 76 extracts the incoming 
digital associated data over the communication line 30 and transfers 
the incoming associated data into the associated data queue 74. 

5 

For one embodiment, the data selector 76 receives packetized 
digital data streams over the communication line 30 from the satellite 
receiver 14. The data selector 76 decodes the header of each incoming 
packet to determine whether to transfer the incoming packet into 
10 either the video queue 70, the audio queue 72, or the associated data 
queue 74. 

For another embodiment, the data selector 76 receives a 
digitized representation of an analog video signal over the 

15 communication line 30 from either the CATV receiver 16 or the 

television broadcast receiver 18. The data selector 76 extracts the video 
information from the incoming digitized video signal and assembles 
the video information into corresponding video packets in the video 
queue 70. The data selector 76 extracts the audio information from the 

20 incoming digitized video signal and assembles the audio information 
into corresponding audio packets in the audio queue 72. The data 
selector 76 extracts associated data during the vertical blanking 
interval of each video frame of the incoming digitized video signal on 
the communication line 30. The data selector 76 assembles the 



04/02/2004, EAST Version: 1.4.1 



PCT7US95/10662 

WO 96/10888 r*.iiiK«:iiiww>* 



-18- 

incoming associated data into associated data packets in the associated 
data queue 74. 

The data selector 76 generates an interrupt to the processor 52 
5 over an interrupt line 78 each time an incoming packet is placed in 
either the video queue 70, the audio queue 72, or the associated data 
queue 74. The interrupt to the processor 52 on the interrupt line 78 
causes the processor 52 to read the incoming packets from the video 
queue 70, the audio queue 72, and the associated data queue 74 over 
10 the system bus 51. The processor 52 men distributes the incoming 
video packets from the video queue 70 to the graphics display 
subsystem 56 to generate a video image in the video window 40. The 
processor 52 distributes the incoming audio packets to the audio 
subsystem 62 to drive the speaker 24. The processor 52 reads the 
15 incoming associated data packets from the associated data queue 74 
and then performs the video command and control functions 
specified in the incoming associated data packets. 

The address filter 75 contains a data stream address for the 
20 computer system 10. The data selector 76 ignores incoming data 

streams mat do not correspond to the data stream address. The data 
stream address may be written by the processor 52. The data stream 
address may specify the same address of the data stream or the 
destination address of the computer system 10 for the data stream. 
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Figure 4 illustrates a digitized representation of an analog video 
signal received by the data selector 76 from either the CATV receiver 
16 or the television broadcast receiver 18 for one embodiment. The 
incoming digitized video signal on the communication line 30 
5 comprises a series of video frames. Each video frame comprises a 
series of scan lines that carries both video and audio information. 
Each video frame further comprises a vertical blanking interval. The 
vertical blanking interval carries the associated data that corresponds 
to the video and audio information for each video frame. 

10 

Figure 5 illustrates the packetized digital data stream received 
by the data selector 76 over the communication line 30 for one 
embodiment. The incoming packetized digital data stream on the 
communication line 30 includes a video packet 80, an audio packet 82, 
15 and an associated data packet 84. 

The video packet 80, the audio packet 82, and the associated data 
packet 84 each comprise a packet header and a packet payload. The 
packet header of the video packet 80, the audio packet 82 and the 
20 associated data packet 84 each include a time stamp (TIME_STAMP) 
that synchronizes the video, audio and associated data carried in the 
packets 80-84. 

The video packet 80 includes a video payload that provides 
25 digital video data for display in the video display window 40. The 
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video packet 80 is identified as a packet that carries video data by the 
video identifier (VIDEOJD) in the packet header. 

The audio packet 82 includes an audio payload for transfer to 
5 the audio subsystem 64 to drive the speaker 24. The audio packet 82 is 
identified as a packet that carries audio data by the audio identifier 
(AUDIOJD) in the packet header. 

The associated data packet 84 includes an associated data 
10 payload that specifies interactive video command and control 

functions for the computer system 10. The associated data packet 84 is 
identified as a packet that carries associated data by the associated data 
identifier (DATAJD) in the packet header. 

15 For other embodiments, the associated data stream is carried 

via a chromanance key, a sideband transmission or an audio sub 
carrier. An embodiment of the command protocol contains 
commands that perform the following functions: 

20 1) Specify coordinate scale 

2) Video screen position and scaling 

3) Screen background color 

4) Screen background pattern 

5) Text , 
25 6) Placement region for text (height and width box) 

7) Font, size and color for text 

8) Graphic/ icon for display 

9) Display graphic/icon at position with scale 

10) Ordering of graphic/text items for layering on the screen 
30 11) Define height and width and position of a selection region 
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12) Define command to be performed when selection region 
selected 

13) Define color palette 

Each command is pre-pended with an address and an identifier. 
The address specifies the address of the receiver (with suitable coding 
for "all" and sub-groups of possible receivers) and the identifier marks 
this as a display command (as opposed to other data that is 
synchronized with the video or audio). 



10 



One possible implementation of the above functions is 
illustrated by the following "object oriented" command set: 
Define Object 

Type: Text, Graphic, Icon, Sound, Cursor, Video 

15 Window, 

Selection Region, Execution Object, Object List 
Content: appropriate to object typo 
Attributes: appropriate to object type: for example, 

text 

20 attributes include font, size, color> 

Instantiate Object 

Background: Transparent, solid color, or repeated object 

(pattern) 

Foreground: Visible/ Invisible 
25 Location: <x, y> 

Size: <x,y> 
Scale: <x,y> 

Define coordinate scaling (low x, high x, low y, high y) 
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Define Virtual Screen (ID, object Ust) 

Display Virtual Screen (ID, on/ off) 

Free Object Instance (ID Ust, <all>) 

Cache Object Instance (ID Ust, <all>) 

5 Define Instance Order (ID Ust) (for overlapping 

objects) 

Move Object Instance (ID, x, y) 

Copy Object Instance (ID, new x, new y) 

10 Figure 6 illustrates the software architecture for the computer 

system 10 for one embodiment. The software for the computer system 
10 includes a client runtime manager 102 and an operating system 100. 
The operating system 100 comprises a set of video driver routines 108, 
a set of audio driver routines 104, a data modem driver 106, a pointing 

15 device driver 110, and a set of display window control routines 112. 

The client runtime manager 102 reads incoming packets from 
the video queue 70, the audio queue 72, and the associated data queue 
74 using the data modem driver 106. The data modem driver 106 

20 manages the video queue 70, the audio queue 72, or the associated 
data queue 74 and processes the interrupts on the interrupt Une 78 to 
the processor 52. The client runtime manager 102 distributes the 
incoming video data from the video queue 70 to the graphics display 
subsystem 56 using the video driver routines 108. The video driver 

25 routines 108 enable the client runtime manager 102 to display the 
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incoming video data stream within the video window 40 on the 
display surface 50 of the display device 12. 

The client runtime manager 102 distributes the incoming audio 
5 data from the audio queue 72 to the audio subsystem 62 using the 
audio driver routines 104. The audio driver routines 104 enable the 
client runtime manager 102 to drive the speaker 24 with the incoming 
audio information from the audio queue 72. 

10 The client runtime manager 102 reads the incoming associated 

data packets from the associated data queue 74 and executes the 
command and control interactive video functions specified by each 
associated data packet. The associated data packets are synchronized to 
the video and audio packets via the TIME_STAMP in each packet 

15 header. The associated data packets from the associated data queue 74 
include commands that specify the placement of graphic objects on 
the display device 12. The associated data packets also include 
commands for placement for graphical windows on the display device 
12. The associated data packets also specify graphical objects for 

20 rendering on the display device 12. 

The associated data packets also specify selection regions for the 
display surface 50 of the display device 12. In addition, the associated 
data packets includes commands for execution upon selection of one 
25 of the specified selection regions by the mouse pointing device 22. 
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The associated data packets also include commands for the 
presentation placement and sizing of the video window 40 on the 
display surface 50. 

5 The display window control routines 112 enable the client 

runtime manager 102 to define the video display window 40 according 
to the specifications provided by the associated data packets. The 
pointing device driver 110 enables the client runtime manager 102 to 
detect the selection of one of the selection regions defined by the 

10 associated data. 

Figure 7 is a flow diagram illustrating the functions of the client 
runtime manager 102 for one embodiment. The client runtime 
manager 102 is driven by interrupts from the data modem 58 over the 

15 system bus 51. At decision block 120, the client runtime manager 102 
determines whether an incoming video packet is available from the 
video queue 70. If an incoming video packet is available at decision 
block 120 then control proceeds to block 122. At block 122 the client 
runtime manager 102 updates the content of the video display 40 

20 using the video driver routines 108 to communicate with the graphics 
display subsystem 56. Thereafter, the client runtime manager 102 
waits for another interrupt from the data modem 58. 

At decision block 124, the client runtime manager 102 
25 determines whether an incoming audio packet is available from the 
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audio queue 72. If an incoming audio packet is available from the 
audio queue 72 then control proceeds to block 126. At block 126 the 
client runtime manager 102 invokes the audio driver routines 104 to 
play out the incoming audio packet to the audio subsystem 62. The 
5 client runtime manager 102 transfers the audio packet payload audio 
data to the audio subsystem 62 to drive the speaker 24. Thereafter, the 
client runtime manager 102 awaits another interrupt from the data 
modem 58. 



10 At decision block 128, the client runtime manager 102 

determines whether an incoming associated data packet is available in 
the associated data queue 74. If an associated data packet is not 
available from the associated data queue 74 then the client runtime 
manager 102 waits the next interrupt from the data modem 58. 

15 

If an associated data packet is available from the associated data 
queue 74 at decision block 128, then control proceeds to block 130. At 
block 130, the client runtime manager 102 reads the associated data 
packet from the associated data queue 74 and performs the interactive 
20 command and control functions specified by the incoming associated 
data packet. Thereafter, the client runtime manager 102 awaits the 
next interrupt from the data modem 58. 
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Figuie 8 illustrates an example arrangement of video and 
display data on the display surface 50 which is generated under 
content programmer control with the techniques described herein. 

5 In the foregoing specification the invention has been described 

with reference to specific exemplary embodiments thereof. It will, 
however, be evident that various modifications and changes may be 
made thereto without departing from the broader spirit and scope of 
the invention as set forth in the appended claims. The specification 

10 and drawings are accordingly to be regarded as illustrative rather thai 
a restrictive sense. 
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What is claimed is: 



1. A method for providing a video display image, comprising the 
5 steps of: 

receiving a video data stream and an associated data stream 
corresponding to the video data stream; 

displaying a video image defined by the video data stream on a 
display device and performing an interactive command function 
10 specified by the associated data stream. 

2. The method of claim 1, wherein the video data stream is 
received during a series of scan intervals of a video frame and the 
associated data stream is received during a vertical blanking interval 

15 of the video frame. 

3. The method of claim 1, wherein the video data stream and the 
associated data stream each comprise a series of digital 
communication packets, each digital communication packet having 

20 an identifier that indicates the video data stream or the associated data 
stream. 

4. The method of claim 1, wherein the interactive command 
function comprises a command that specifies a set of parameters that 
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determines an area on a display surface of the display device for 
placement of a video display window that contains the video image. 

5. The method of claim 1, wherein the interactive command 

5 function comprises a command that specifies a set of parameters that 
determines an area on a display surface of the display device for 
placement of a graphical object that corresponds to the video image. 

6. The method of claim 5, wherein the interactive command 
10 function comprises a command that specifies a set of pixel data or 

graphical description commands that correspond to the graphical 
object. 

7. The method of claim 1, wherein the interactive command 

15 function comprises a command that specifies a set of parameters that 
determines an area on a display surface of the display device for 
placement of a selection window that corresponds to the video image. 

8. The method of claim 7, wherein the interactive command 

20 function comprises a command that specifies an interactive command 
that is performed if a user selects the selection window. 

9. A method for creating a display in a computer system, 
comprising the steps of: 
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receiving a video stream and a data stream synchronized to the 
video stream, the data stream specifying at least one graphical 
command; 

generating a video scene defined by the data stream onto a 
5 portion of a display screen of the computer system; 

performing a graphical operation on the computer screen 
defined by the command. 

10. The method of claim 9, further comprising the steps of 

10 receiving an audio stream synchronized to the video stream and 
playing the audio stream through an audio subsystem of the 
computer system. 

11. The method of claim 9, wherein the video stream is coded in a 
15 series of video scan intervals of a video signal and the data stream is 

coded in a series of nonvideo scan intervals of the video signal. 

12. The method of claim 9, wherein the data stream specifies a 
graphical object for display on the display screen. 

20 

13. The method of claim 9, wherein the graphical command 
specifies a color palette for the display screen. 

14. The method of claim 9, wherein the graphical command 
25 specifies placement of a graphical object on the display screen. 
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15. The method of claim 9, wherein the graphical command 
specifies a set of parameters that define selection regions on the 
display screen. 

16. The method of claim 15, wherein the graphical command 
specifies a selection device for picking the selection regions on the 
display screen. 

17. The method of claim % wherein the graphical command 
specifies text for display on the display screen. 



18. The method of claim 17, wherein the graphical command 
specifies placement and format of the text including font, color, and 

15 point size. 

19. The method of claim 9, wherein the data stream comprises a 
series of data packets and wherein the step of receiving a video stream 
and a data stream synchronized to the video stream includes the step 

20 of filtering the data packets according to a destination address of each ■ 
data packet. 

20. The method of claim 9, wherein the data stream comprises a 
series of data packets and wherein the step of receiving a video stream 

25 and a data stream synchronized to the video stream includes the step 
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of filtering the data packets according to a source address of each data 
packet. 
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